Finding structure in text, genome and other symbolic sequences

Acknowledgments I would like to particularly thank my advisor, Yorick Wilks. Over the years I have known him, Yorick has always been insightful, cheerful and generous. His ability to understand and constructively critique research over an extraordinarily wide range of topics has been extremely helpful, both professionally and academically. Moreover, his willingness to consider and even nurture points of view which run counter to his own intuitions is extraordinary in a world filled with entirely too many ideologues. Even more extraordinary is his willingness to adopt techniques that have proven useful regardless of his initial assessment of them. This flexibility is the mark of a true scientist. The members of my committee, Peter Willett, Robert Gaizauskis and Steve Renals, also provided valuable advice and assistance and oversight. Peter has been especially helpful by sharing his wide knowledge of and historical perspective on the field of information retrieval. In addition, he has been willing to carefully edit this work to graciously help me to completely avoid the use of split infinitives. Robert has spent more time helping me and examining this work than could ever have been expected. Outside of the university, Ellen Friedman also provided enormous support in the form of advice and careful review of this thesis. Her guidance was critical to the quality of the chapters on genomic sequence analysis. Her advice and insight throughout this work have helped me make this thesis far better than it otherwise could have been. Jamie Callan reviewed the chapter on document routing and provided helpful comments. At least as important, however, he supervised the preparation of the InRoute comparison results which provided a connection between my results in in chapter 7 and the IR literature in general. Owen White extracted and provided data for the genomic sequence analysis chapters. Only those who have worked with the sequence databases can know how helpful this was. My current employer, Aptex Software, and the parent company, HNC Software, are to be commended for their encouragement and support during the preparation of this thesis. They have provided financial support, access to computational resources and the Convectis system. This support has been crucial to the completion of this thesis. Finally, my examiners, Keith van Rijsbergen and Steve Renals, are taking a substantial amount of their time to judge this work. I thank them. Summary The statistical methods derived and described in this thesis provide new ways …